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1.  Overview 


The  Defense  Advanced  Research  Project  Agency  (DARPA)  and  the  Joint  Forces  Command 
(JFCOM)  developed  a  suite  of  decision  support  tools  to  enhance  the  capability  of  commanders 
and  staffs  to  plan  and  execute  effects-based  campaigns.  The  complex  environments  in  which 
commanders  and  staffs  must  operate  brings  in  other  elements  besides  military  issues — such  as 
political,  economic,  social,  information,  and  infrastructure  (PMESII).  In  order  to  better  assess, 
understand,  and  explore  the  relevant  and  important  issues  across  these  operational  dimensions, 
DARPA  and  JFCOM  have  developed  the  Conflict  Modeling,  Planning,  and  Outcomes 
Experimentation  (COMPOEX)  program  tool  suite  to  support  the  commander  and  staffs  (Corpac 
et  al.,  2006).  Across  the  development  of  this  tool  suite,  multiple  limited  objective  experiments 
(LOEs)  were  conducted  to  test  and  assess  the  progress  and  effectiveness  of  one  or  more  of  the 
tools.  This  report  describes  the  LOE  addressing  the  Campaign  Planning  Tool-Light  (CPT-L) 
and  its  results. 

The  COMPOEX  LOE  2-1  was  designed  to  test  how  well  the  CPT-L  was  able  to  capture  a  plan 
through  a  computer-mediated  tool  without  providing  additional  workload  and  requirements  on 
the  users.  The  primary  objective  was  examined  through  an  analysis  of  both  the  planning  process 
and  the  planning  product  produced  within  the  one-day  event.  The  results  in  the  areas  of  planning, 
setting  objectives  and  alternatives,  and  situation  understanding  show  that  the  CPT-L  was  as 
effective — if  not  more  effective — in  facilitating  the  planning  process  compared  to  a  team  using 
more  traditional  tools  and  methods.  The  quality  of  the  planning  product  produced  was  not  able  to 
be  fairly  compared  due  to  several  unanticipated  confounds  of  the  experiment  directly  affecting 
them  and  therefore  creating  an  unfair  comparison.  None  of  the  results  alone  provide  definitive 
results,  but  the  converging  nature  of  the  planning  process  data  all  in  the  same  direction  suggests 
a  beneficial  role  of  the  CPT-L  in  the  planning  process.  In  order  to  effectively  assess  the  impact 
of  the  CPT-L  on  the  product  of  the  planning  process,  additional  training  for  CPT-L  users  is 
required  to  create  equal  expertise  with  the  tools  being  used  across  both  teams  and  equal  amounts 
of  planning  time  and  manpower  available  is  also  required. 


2.  Introduction 


In  order  to  explore  and  consider  the  full  range  of  military  and  nonmilitary  capabilities  across 
different  levels,  a  much  broader  approach  that  considers  other  elements  other  than  military  issues 
must  be  taken.  The  PMESII  elements  related  to  the  region  and  situation  need  to  be  considered  by 
the  commander  and  staff  during  planning  and  mission  preparation.  In  order  to  facilitate  the 
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consideration  of  a  much  wider  realm  of  issues,  effects,  and  implications,  DARPA  and  the 
JFCOM  collaborated  in  the  design  of  the  COMPOEX  tool  suite.  One  of  the  tools  intended  to 
support  and  augment  PMESII  planning  is  the  CPT-L. 

The  CPT-L  was  designed  to  augment  strategic  planning  by  creating  an  easy  to  use  interface  that 
promotes  collaboration  across  lines  of  efforts  within  the  planning  framework.  The  CPT-L  tool 
allows  the  planner  to  define  relevant  PMESII  actions  and  effects  across  various  courses  of 
actions  if  there  are  more  than  one  (e.g.,  governance,  economic,  rule  of  law).  The  interface  is 
similar  to  Microsoft  Project*  as  actions  and  effects  are  placed  on  a  user-defined  timescale  with 
durations  identified  for  each  action  (figure  1).  The  specific  CPT-L  capabilities  designed  to 
improve  planning  are: 

1.  A  semistructured  environment  for  planning  tasks  (e.g.,  drawing  lines  of  effort,  defining 
objectives,  defining  durations,  defining  interdependencies). 

2.  Ability  to  visualize  and  merge  multiple  lines  of  effort. 


Figure  1.  Example  of  the  CPT-L  interface. 

These  capabilities  are  hypothesized  to  positively  influence  several  general  functions  of 
Command  and  Control  (C2)  or  mission  command  identified  by  Van  Creveld  (1985),  which  have 


* 


Microsoft  Project  is  a  trademark  of  Microsoft  Corporation. 
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also  been  tied  to  the  theoretical  distributed  cognition  framework  (Hansberger,  2008).  The 
semistructured  environment  is  hypothesized  to  improve  performance  by  facilitating  the  user’s 
ability  to  manipulate  data  and  therefore  reduce  workload.  The  ability  to  visualize  and  merge 
multiple  lines  of  effort  is  hypothesized  to  improve  identifying  objectives  and  working  out 
alternative  means  for  obtaining  those  objectives  along  with  improving  the  planning  teams’ 
situational  understanding.  Both  capabilities  are  hypothesized  to  positively  influence  the  actual 
planning  process  (figure  2).  Three  of  the  four  identified  C2  functions  were  tested  in  this  study 
including  data  manipulation,  planning,  and  situation  understanding. 


Figure  2.  CPT-L  capabilities  mapped  to  general  C2  functions. 


3.  Method 


3.1  Participants 

There  were  a  total  of  1 1  participants  for  LOE  2-1.  They  consisted  of  four  Senior  (Sr.)  Mentors 
and  seven  active  military  planners  and  were  divided  into  two  teams  of  five  and  six  members  with 
a  supervisor  over  both  groups.  The  two  groups  represented  the  experimental  (team  with  the  CPT) 
and  control  (team  without  the  CPT)  groups.  The  two  groups  were  designed  to  have  an  equal 
number  of  members;  however,  one  member  of  the  experimental  team  was  not  able  to  attend  and 
there  was  not  adequate  time  to  obtain  a  replacement.  Team  members  were  divided  across  the 
teams  to  balance  domain  expertise,  quantity  of  experience,  and  Sr.  Mentor/Planner  status. 

Among  the  Sr.  Mentors  for  each  team,  one  had  an  extensive  military  background  and  the  other 
had  an  extensive  government  (non-Department  of  Defense)  background. 


3 


3.2  Materials 

Each  team  had  access  to  six  computers  with  Microsoft  Office*  (including  MS  Project)  installed, 
Internet  access  for  the  CPT-L  team  only  to  enable  the  CPT-L  (no  access  to  external  Internet 
sites),  and  a  projector  connected  to  one  of  the  six  computers.  There  were  whiteboards  and 
corkboards  provided  for  the  non-CPT-L  team  along  with  Post-it*  notes.  Each  team  was  located 
in  separate  rooms.  A  digital  camera  was  used  to  capture  whiteboard  drawings  and  use  of  the 
corkboards.  The  pictures  were  used  by  the  researchers  to  analyze  the  planning  process  and 
outcomes. 

3.3  Design  and  Procedure 

An  experimental  versus  control  group  design  was  implemented.  The  manipulation  was  the 
planning  tools  provided  to  each  team,  the  experimental  team  had  the  CPT-L  and  the  control  team 
did  not  have  the  CPT-L.  The  dependent  variables  examined  include  variables  for  planning,  data 
manipulation,  situation  understanding,  and  usability.  Training  developed  by  BAE  Systems  on  the 
CPT-L  was  conducted  at  the  beginning  of  Part  I  for  the  CPT  group  and  at  the  end  of  Part  I  for 
the  non-CPT  group. 

The  training  session  lasted  for  15  minutes  (min)  and  was  presented  to  the  CPT-L  team  as  a  slide 
presentation  without  active  participation  by  the  CPT-L  members.  Additional  time  that 
approached  45  min  with  the  CPT-L  developers  was  required  before  the  CPT-L  members  felt 
comfortable  with  the  tool  and  began  entering  planning  data.  This  occurred  at  the  beginning  of  the 
planning  task. 

Each  planning  team  was  asked  to  review  read-ahead  materials  that  describe  the  Chad  scenario 
situation  and  background  information.  They  were  given  a  brief  overview  of  the  experiment  plan 
during  the  orientation  (figure  3).  In  the  read-ahead  materials,  the  team  members  received  the 
policy  aims,  interim  states  to  be  achieved,  four  lines  of  effort,  and  some  pre-established 
objectives  and  effects.  The  overall  planning  task  was  to  begin  planning  for  a  comprehensive 
strategy  of  transformation  across  four  lines  of  effort  (political,  security,  rule  of  law,  and  political 
economic),  which  lasted  for  4  hours  (h)  and  15  min  and  extends  across  the  CPT/Control  Planning 
and  Transference  and  Combine,  Sequence,  and  Deconflict  portions  of  the  timeline  (figure  3).  The 
plenary  session  at  the  end  involved  both  groups  discussing  their  experiences  with  and  without  the 
tools. 


^Microsoft  Office  is  a  trademark  of  Microsoft  Corporation. 
*Post-it  is  a  registered  trademark  of  3M. 
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Figure  3.  LOE  day  one  timeline. 

3.4  Planning  Measures 

3.4.1  Observer  Ratings  (Product  Focus) 

The  final  plans  from  both  teams  were  presented  to  two  subject  matter  experts  (SMEs)  in  the 
same  format.  The  SMEs  evaluated  the  two  plans  while  blind  to  the  experiment  conditions.  The 
SMEs  evaluated  the  plans  across  six  planning-relevant  dimensions  that  included:  (1) 
Consistency,  (2)  Robustness,  (3)  Conflicts,  (4)  Comprehensiveness,  (5)  Probability  for  plan 
success,  and  (6)  Plan  Quality  (see  appendix). 

3.4.2  Social  Network  Analysis 

Social  network  analysis  is  based  on  network  theory,  which  uses  graphs  as  a  representation  of 
symmetric  or  asymmetric  relations  between  discrete  objects  (Scott,  2000).  The  graph  is  a 
mathematical  structure  to  represent  pairwise  relations  between  objects.  Placed  within  a  social 
context  of  humans  and  their  interactions,  a  social  network  is  a  set  of  individuals  (i.e.,  nodes) 
connected  through  social  interactions  such  as  face-to-face  or  email  communication  (i.e.,  links). 
The  analysis  of  these  social  networks  consists  of  a  family  of  relational  methods  to  systematically 
uncover  patterns  of  interconnectedness. 

3.5  Data  Manipulation  Measure 
3.5.1  Workload 

The  CPT-L  was  designed  to  modify  the  way  planners  interact  and  manipulate  information  and 
plan  relevant  data  through  its  semistructured  planning  environment.  The  CPT-L  interface  was 
hypothesized  to  reduce  the  workload  of  the  participants  using  the  tool.  Members  of  both  teams 
completed  the  National  Aeronautics  and  Space  Administration  (NASA)  Task  Load  Index  (TLX) 
subjective  evaluation  of  their  workload  after  their  planning  activities  (Hart  and  Staveland,  1988). 
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NASA  TLX  is  a  subjective  workload  assessment  tool  that  allows  users  to  perform  subjective 
workload  assessments  on  operator(s)  working  with  various  human-machine  systems.  NASA 
TLX  is  a  multidimensional  rating  procedure  that  derives  an  overall  workload  score  based  on  a 
weighted  average  of  ratings  on  six  subscales.  This  can  be  completed  in  a  short  amount  of  time 
through  a  simple  computer  program. 

•  “Mental  demand”  refers  to  how  much  mental  and  perceptual  activity  was  required 
(thinking,  deciding,  calculating,  remembering,  looking,  searching,  etc.)  during  the  task.  The 
respondent  should  consider  whether  the  task  was  easy  or  demanding,  simple  or  complex,  or 
exacting  or  forgiving. 

•  “Physical  demand”  measures  the  required  physical  activity  in  relation  to  whether  the  task 
was  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  or  restful  or  laborious. 

•  The  amount  of  experienced  time  pressure  is  measured  by  the  “temporal  demand”  subscale. 
It  addresses  issues  such  as  whether  the  pace  of  interaction  was  slow  and  leisurely,  or  rapid 
and  frantic. 

•  “Performance”  refers  to  how  successful  respondents  think  they  were  in  accomplishing  the 
goals  of  the  task  set  by  the  experimenter,  and  how  satisfied  they  were  with  their 
performance  in  accomplishing  these  goals. 

•  The  criteria  of  “effort”  requests  the  respondents  to  assess  how  hard  they  had  to  work 
(mentally  and  physically)  to  accomplish  the  level  of  performance  they  achieved. 

•  Finally,  evaluation  of  the  “frustration”  level  measures  how  insecure,  discouraged,  irritated, 
stressed,  and  annoyed  versus  secure,  confident,  relaxed,  and  complacent  subjects  felt 
during  the  task. 

3.6  Situation  Understanding  Measure 

3.6.1  Pathfinder 

The  visualization  capabilities  of  the  CPT-L  were  intended  to  improve  the  awareness  and 
understanding  of  its  users  as  it  displays  multiple  LOEs  in  a  chronological  fashion  with 
interdependencies  across  LOEs.  The  Pathfinder  methodology  was  used  to  evaluate  the 
progression  of  understanding  for  both  teams  from  pre  to  postplanning.  It  was  also  used  to 
measure  the  degree  of  shared  understanding  among  the  members  of  each  team  and  how  that 
changed  over  time. 

3.6.2  Pathfinder  Coherence 

One  of  the  Pathfinder  measures  employed  is  the  measure  of  data  coherence.  The  coherence 
measure  reflects  the  consistency  of  the  data.  Coherence  is  based  on  the  assumption  that 
relatedness  between  a  pair  of  items  can  be  predicted  by  the  relations  of  the  items  to  other  items  in 
the  set  (Interlink,  2004).  Past  research  has  found  that  the  coherence  measure  often  correlates  with 
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expertise  or  degree  of  learning  if  examined  over  time  (e.g.,  Housner  et  al.,  1993;  Gualtieri  et  al., 
1996). 

3.6.3  Pathfinder  Networks 

The  Pathfinder  networks  (PFnets)  allow  for  the  visual  representation  of  the  associative  network 
among  the  situational  concepts  in  a  network  form.  The  PFnets  represent  an  individual’s 
understanding  or  situational  model  of  the  targeted  concepts  that  were  included  in  the  Pathfinder 
rating  process. 

There  were  two  primary  confounds  that  developed  within  the  experiment  that  could  influence  the 
results  when  the  two  groups  are  compared.  The  first  confound  is  the  unequal  participant  numbers 
between  the  groups  of  five  for  the  CPT-L  team  and  six  for  the  non-CPT-L  team.  The  reduction 
of  16.7%  manpower  for  the  CPT-L  team  put  that  group  at  an  immediate  disadvantage.  The 
second  confound  is  what  resulted  in  unequal  planning  times  between  the  two  teams.  Even  though 
the  same  amount  of  time  was  allotted  for  both  teams,  the  additional  training  time  (approximately 
45  min  or  approximately  17.7%  of  total  planning  time)  required  by  the  CPT-L  team  and  two 
instances  of  lost  data  by  the  CPT-L  team,  led  to  approximately  about  90  min  of  one  team 
member’s  time,  which  put  the  CPT-L  team  at  an  even  greater  disadvantage. 


4.  Results  and  Discussion 


4.1  Missing  Data 

One  participant  in  the  non-CPT-L  team  had  missing  data  for  the  Pathfinder  and  workload 
measures  for  unknown  reasons.  However,  technical  issues  with  the  computer  used  for  this 
participant  is  suspected  as  the  cause  of  this  missing  data.  Two  additional  non-CPT-L  members’ 
data  was  excluded  from  analysis.  The  non-CPT-L  leader’s  Pathfinder  data  was  excluded  due  to 
very  low-coherence  scores  pre  and  post  and  a  strong  decline  in  coherence  from  pre  to  post.  The 
coherence  measure  reflects  the  consistency  of  the  data.  The  coherence  of  a  set  of  data  is  based  on 
the  assumption  that  relatedness  between  a  pair  of  items  can  be  predicted  by  the  relations  of  the 
items  to  other  items  in  the  set.  Past  research  has  indicated  that  coherence  scores  of  less  than  0.20 
suggest  the  participant  did  not  take  the  task  seriously  or  there  was  some  error  in  the  input  of  the 
Pathfinder  ratings.  The  coherence  score  for  this  individual  was  0.12  and  -0.13,  respectively,  for 
pre  and  postscores. 

Another  of  the  non-CPT-L  team  member’s  data  was  excluded  from  both  the  Pathfinder  and 
Workload  analysis.  In  addition  to  extremely  low-coherence  on  the  Pathfinder  pretest  (-0.12), 
there  was  no  score  for  the  post-test  due  to  the  same  value  entered  for  each  of  the  135  ratings,  and 
observational  data  that  the  participant  did  not  take  the  measure  task  seriously.  This  individual’s 
workload  data  was  more  than  two  standard  deviations  lower  than  the  group’s  and  was  also 
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excluded  from  analysis.  It  is  recommended  that  future  participation  by  this  person  is 
accompanied  by  clear  instructions  on  the  importance  of  the  measures  and  close  monitoring  of  his 
responses. 

Overall,  there  were  three  members  of  the  non-CPT-L  group  who  were  missing  or  excluded  from 
the  Pathfinder  analysis  (n  =  3).  There  were  two  members  of  the  non-CPT-L  group  who  were 
missing  or  excluded  from  the  workload  analysis  (n  =  4). 

4.2  Planning 

Planning  was  evaluated  through  three  analysis  techniques.  Two  of  these  techniques  focused  on 
the  product  of  the  planning  process,  while  the  third  focused  on  the  planning  process  itself. 

4.3  Plan  Depth  (Planning  Product) 

Depth-of-plan  was  defined  by  the  number  of  objectives,  effects,  actions,  resources, 
interdependencies,  and  nodes  identified  in  each  of  the  final  plans.  Each  planning  element  was 
given  a  weight  of  one,  with  the  exception  of  the  interdependencies.  Interdependencies  were 
given  a  weight  of  two,  in  order  to  represent  the  higher  level  of  importance  of  the 
interdependencies  to  the  overall  plan. 

The  non-CPT-L  plan  was  entered  into  the  tool  in  order  to  assess  the  depth  of  both  plans  using 
the  same  criteria  mentioned  above.  The  overall  depth  scores  were  practically  the  same  (CPT-L  = 
97;  non-CPT-L  =  94)  between  the  two  teams.  However,  the  distribution  of  the  various  elements 
is  different,  particularly  among  the  number  of  objectives,  effects,  and  inter-LOE  dependencies 
(i.e.,  dependencies  across  LOEs)  identified  in  their  plans  (figure  4).  The  CPT-L  team  identified 
more  effects  and  inter-LOE  dependencies,  while  the  non-CPT-L  team  identified  more  objectives 
in  their  planning. 
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Figure  4.  Plan  depth  across  plan  elements  between  the  CPT-L  and  non-CPT-L  teams. 

4.3.1  Observer  Ratings  (Planning  Product) 

The  SMEs  evaluated  the  plans  across  six  planning-relevant  dimensions  that  included:  (1) 
Consistency,  (2)  Robustness,  (3)  Conflicts,  (4)  Comprehensiveness,  (5)  Probability  for  plan 
success,  and  (6)  Plan  Quality  (see  appendix).  Intraclass  correlation  was  used  as  a  measure  of 
inter-rater  reliability  (IRR)  between  the  evaluators  and  displayed  a  lower  than  desired  value  of  r 
=  0.42  (where  0.70  is  acceptable).  There  was  a  significant  difference  in  the  evaluator  mean 
scores  across  the  six  dimensions;  evaluator  one  had  a  mean  score  of  3.1  and  evaluator  two  had  a 
mean  score  of  2.1  ( t  =  2.19,  p  =  0.01)  on  a  five-point  scale.  Future  evaluators  should  be  involved 
in  IRR  training  to  increase  their  IRR  skills.  Both  evaluators,  however,  showed  the  same  basic 
pattern  of  evaluations  across  the  two  groups,  rating  the  non-CPT-L  group  equal  or  higher  for 
each  planning  dimension  and  overall. 

The  mean  score  across  both  SME  evaluators  showed  a  higher  rated  evaluation  for  the 
non-CPT-L  group  (CPT-L  =  2.0,  non-CPT-L  =  3.1  ,t  =  -3.4, p  <  0.01).  Figure  5  shows  the 
evaluation  means  across  both  SMEs.  Figure  6  shows  the  mean  evaluator  scores  broken  down  by 
each  planning  dimension  and  displays  consistently  higher  evaluated  scores  for  the  non-CPT-L 
plan. 
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Figure  5.  Overall  mean  evaluations  across  both  SME  evaluators  with  a  significant  difference 
between  the  two  groups. 


Figure  6.  Mean  evaluation  scores  broken  out  by  each  plan  dimension. 
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Follow-up  interviews  were  done  with  each  of  the  SME  evaluators  to  provide  further  explanation 
of  the  evaluations.  The  evaluators  were  fairly  consistent  in  their  impressions  of  each  of  the  plans. 
Both  evaluators  agreed  that  the  non-CPT-L  plan  had  a  greater  level  of  detail  that  was  provided 
through  explanatory  text  embedded  in  each  of  the  planning  components  (i.e.,  actions,  effects,  and 
objectives).  The  additional  planning  component  text  was  represented  in  four  different  areas  for 
each  planning  component:  (1)  Resources,  (2)  Assumptions,  (3)  Metrics,  and  (4)  Description.  The 
evaluators  felt  that  the  greater  amount  of  explanation  in  these  areas  allowed  a  better 
understanding  of  the  planners’  intentions  and  allowed  the  identification  of  a  consistent  thread 
throughout  the  non-CPT-L  plan  that  was  not  evident  in  the  CPT-L  plan. 

An  examination  of  the  explanation  text  that  was  included  in  each  of  the  two  plans  was  conducted 
to  further  explore  the  comments  from  the  SME  evaluators.  A  striking  difference  was  evident  in 
the  amount  of  text  present  in  each  of  the  two  plans  (figure  7).  The  non-CPT-L  plan  had 
dramatically  more  text  in  every  category  and  528%  more  text  overall.  This  can  be  directly 
attributable  to  the  differences  the  evaluators  observed  between  the  two  plans. 


Amount  of  Explanation  Text  in  Plans 

1600 


■  Resources  Assumptions  ■  Metrics  ■  Description 


Figure  7.  Amount  and  type  of  accompanying  text  embedded  in  the  planning  components 
for  each  plan. 

Both  evaluators  saw  merit  in  the  CPT-L  plan  and  did  not  see  any  strategic  or  general  approach 
deficiencies.  When  asked  how  the  CPT-L  plan  could  be  improved,  one  evaluator  suggested, 
“allotting  more  time”  in  order  to  provide  the  detail  found  in  the  non-CPT-L  plan.  The  inclusion 
of  explanatory  text  to  each  planning  component  appears  to  be  heavily  dependent  on  the  time  and 
manpower  available.  The  CPT-L  team  had  confounds  working  against  both  of  these  factors  that 


11 


impaired  at  least  20%  of  the  overall  planning  time  available  and  reduced  the  manpower  of  the 
CPT-L  by  17%  compared  to  the  non-CPT-L  team.  Considering  that  the  confounds  directly 
affected  the  factors  necessary  to  provide  the  level  of  detail  found  in  the  non-CPT-L  plan,  a  clear 
conclusion  on  the  quality  of  the  two  plans  and  the  differences  between  them  cannot  be  made 
based  on  this  data.  Since  there  is  not  a  fair  comparison  among  the  planning  product  of  the  teams, 
further  examination  of  the  process  of  planning  and  intermediate  products  (e.g.,  mental  models) 
will  be  examined. 

4.3.2  Social  Network  Analysis  (Planning  Process) 

Social  network  data  was  collected  by  observation.  The  observations  recorded  two  types  of 
collaborative  interactions:  (1)  when  individuals  spoke  face-to-face  as  dyads  or  subgroups  and  (2) 
when  individuals  spoke  to  the  team  as  a  whole.  Additionally,  observations  recorded  the  length  of 
each  interaction,  which  provides  a  means  for  assessing  the  strength  of  the  interactions  between 
dyads  and  for  assessing  the  amount  of  face-to-face  collaboration  that  took  place.  The  data  was 
coded  in  the  following  manner  to  produce  distinct  categories  for  strength  of  interaction  tie 
between  each  dyad  or  pairs  of  individuals:  0  min  =  0,  1-10  min  =1,1 1-20  min  =  2,  and  so  on.  A 
higher  coded  number  relates  to  a  stronger  interaction. 

Figure  8  shows  the  dyadic  and  subgroup  coded  interactions  by  team.  CPT-1  and  non-CPT-7  are 
the  leaders  of  each  respective  group.  The  line  weights  represent  the  amount  of  interactions 
between  any  two  individuals.  The  interaction  between  CPT-2  and  CPT-6  is  very  strong,  as  these 
individuals  worked  together  on  the  same  line  of  effort.  The  CPT-L  team’s  verbal  communication 
structure  is  very  centralized  (around  the  leader,  CPT-1)  and  is  typically  very  efficient  for  routine 
problems  (Shaw,  1964).  It  was  expected  that  the  amount  of  overall  CPT-L  communications 
would  be  less  compared  to  the  non-CPT  team  communication,  but  for  the  CPT-L  team  structure 
to  resemble  more  of  a  fully  connected  network  structure — this  was  not  the  case.  There  are  a 
couple  of  possible  explanations  for  this  result. 

1 .  The  CPT  team  spent  considerable  time  trying  to  understand  and  use  the  tool.  The  time 
spent  by  team  members  interacting  with  the  software  developers  may  have  limited  cross¬ 
functional  interactions. 

2.  The  tool  supported  collaborative  activities,  which  alleviated  the  need  for  face-to-face 
interactions  across  functions. 
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Figure  8.  Dyadic  and  subgroup  coded  interactions  by  team. 

The  non-CPT-L  team  displayed  a  fully  connected  verbal  communication  structure  that  is 
typically  conducive  for  complex  tasks  that  require  extensive  collaboration  (Shaw,  1964).  The 
strong  interactions  found  between  non-CPT-7  and  non-CPT-12  along  with  non-CPT-10  and  non- 
CPT-1 1  represent  pairs  of  individuals  working  on  the  same  LOE  together.  The  difference  in  the 
communication  structures  also  is  evident  in  the  density  of  each  team’s  networks.  The  difference 
in  density  between  the  two  teams  (CPT-L  team  =  0.4,  non-CPT-L  team  =  1.0)  indicates  a  much 
higher  level  of  verbal  communications  among  the  non-CPT-L  team. 

The  higher  level  of  verbal  communications  by  the  non-CPT-L  team  was  only  evident  in  the 
dyadic  interactions  between  team  members.  When  team  interactions  are  included,  the  CPT-L 
team  displays  a  greater  amount  of  time  spent  for  team  interactions  (figure  9).  This  could  be  due 
to  the  CPT-L  providing  a  big-picture  to  all  team  members,  which  allowed  them  more  time  to 
discuss  planning  issues  as  a  group.  For  example,  the  non-CPT-L  group  had  just  finished  putting 
together  a  graphic  combining  the  individual  LOEs  by  experiment  end.  This  did  not  allow  them 
much  time  to  discuss  the  big-picture  and  draw  dependencies.  In  contrast,  the  CPT-L  team  was 
able  to  reach  an  integrated  planning  state  that  considered  the  input  of  each  of  the  LOEs  earlier 
and  allowed  a  focus  on  the  dependencies  among  individual  LOEs. 
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Figure  9.  Total  minutes  of  team  and  dyad  interactions  by  team. 


Further  analysis  into  the  team  interactions  shows  the  individual  contributions  in  the  team 
interactions  (figure  10).  As  expected,  the  formal  leaders  of  each  team  dominated  the  discussion 
time.  Contributions  were  made  by  other  team  members  as  depicted  in  figure  10. 


Figure  10.  Total  minutes  within  team  interactions  by  individual  team  members. 
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4.3  Data  Manipulation 


The  workload  results  across  both  teams  show  no  differences  except  for  the  area  of  physical 
demand  (figure  11).  The  CPT-L  team  showed  significantly  less  physical  demand  compared  to 
the  non-CPT-L  team,  although  both  teams  reported  a  low  amount  of  physical  demand  overall.  It 
is  worth  noting  that  even  with  the  need  for  extended  training  and  some  loss  of  data,  there  was  no 
significant  difference  between  the  two  groups  on  the  frustration  dimension.  Within  the  CPT 
however,  there  were  a  few  individuals  that  reported  a  high  level  of  frustration.  When  divided  by 
age  groups  consisting  of  a  below  60  group  and  a  60  and  above  group,  there  was  a  large  difference 
between  the  two  groups  (below  60  =  23.3;  above  60  =  67.5).  Even  with  such  few  participants,  this 
strong  correlation  (r  =  0.87)  between  age  and  frustration  was  significant  (t  =  3.09,  p  =  0.05).  This 
finding  highlights  an  age  difference  of  comfort  and  familiarity  with  computer  tools  and 
interfaces  that  can  exist  between  generations  or  age  groups  (Borghans  and  Weel,  2002).  In  such 
cases,  the  older  population  does  not  have  the  same  familiarity  with  such  tools  and  therefore 
experiences  more  frustration  with  their  use. 


Figure  11.  NASA  TLX  workload  results  for  both  the  CPT-L  and  non-CPT-L  teams.  The  difference  in  the  Physical 
Demand  is  the  only  significant  difference. 

4.4  Situation  Understanding 

4.4.1  Pathfinder 

The  Pathfinder  metric  (Schvaneveldt,  1990)  elicits  associative  memory  networks  and  provides 
information  structures  revealing  the  underlying  organization  of  the  data.  These  networks  can  be 
statistically  compared  across  time  with  each  other  or  against  a  benchmark  network.  Pathfinder 
data  is  collected  by  asking  the  individual  to  provide  a  series  of  relational  scores  across  the 
targeted  topics  and  issues  (figure  12).  Seventeen  concepts  (table  1)  were  used,  which  created  136 
ratings  between  all  possible  pairings  of  the  concepts.  The  resulting  information  structures  can 
then  be  visualized  and  empirically  analyzed. 
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Figure  12.  Pathfinder  rating  task. 

Table  1.  Pathfinder  scenario-specific  concepts. 


Pathfinder  Concepts 

Political  line  of  effort 

Darfur  rebels 

Security  line  of  effort 

UN  Coalition 

Rule  of  law  line  of  effort 

Oil  revenue 

Political  economic  line  of  effort 

Regional  stability 

World  Bank 

Economic  growth 

Sudan 

Civil  war 

President  Deby 

Legitimate  government 

Zeghawa 

President  Chiroc 

Ethnic  tension 

— 

4.4.2  Coherence  Measure 

The  premeasure  of  coherence  for  the  CPT-L  and  non-CPT-L  teams  shows,  as  expected, 
approximately  the  same  level  of  situational  expertise.  However,  the  trend  from  pre  to 
postplanning  shows  greater  learning  achieved  by  the  CPT-L  group  (figure  13),  with  little 
improvement  evident  among  the  non-CPT-L  team.  Although  a  larger  sample  is  needed  to  test  for 
statistical  significance,  this  finding  suggests  that  the  CPT-L  facilitated  learning  of  the  situation 
and  environment  in  spite  of  confounds  mentioned  above  that  put  the  CPT-L  at  a  disadvantage 
(i.e.,  one  less  member  and  technical/training  issues).  Figure  14  highlights  the  slightly  higher 
average  coherence  score  for  the  CPT-L  team  at  the  2nd  or  postmeasure  of  coherence. 
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Figure  13.  Average  coherence  scores  over  time  representing 
the  amount  of  learning  achieved  by  each  team. 


Figure  14.  Average  postcoherence  scores  between  the  CPT-L 
and  non-CPT-L  teams. 


4.4.3  Pathfinder  Networks 

The  PFnets  can  be  quantitatively  compared  with  one  another  to  evaluate  the  degree-of-similarity 
between  them  and  therefore  evaluate  the  degree-of-similarity  of  one  individual’s  or  group’s 
understanding  to  another.  The  similarity  measure  compares  the  number  of  links  in  common 
between  two  PFnets  (Interlink,  1994).  Identical  networks  provide  a  similarity  score  of  one,  while 
networks  that  share  no  links  provide  a  similarity  score  of  zero.  The  comparison  of  the  pre-PFnets 
for  both  teams  show  a  small  amount  of  similarity  (24%)  between  the  two  teams  (figure  15a  and 
15b).  An  examination  of  the  post-PFnets  for  both  teams  shows  a  large  increase  between  the  two 
teams  at  44%  (figure  16a  and  16b).  This  increase  suggests  that  even  though  the  two  teams  were 
using  different  tools  in  the  planning  process,  their  understanding  or  perception  of  the  situation  is 
gaining  in  similarity  and  on  a  path  of  convergence. 
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Examination  of  the  PFnets  within  the  same  team  from  pre  to  postmeasurements  gives  an 
indication  of  the  degree-of-change  within  that  team’s  perception  of  the  situation  from  before  and 
after  the  planning  period.  The  CPT-L  team  had  a  similarity  score  of  54%  (figures  15a  and  16a) 
from  its  pre  and  postmeasurements,  while  the  non-CPT-L  team  had  a  similarity  score  of  13% 
(figures  15b  and  16b).  These  results  suggest  that  there  was  only  moderate  change  within  the 
CPT-L  team  from  pre  to  post  while  the  non-CPT-L,  with  a  very  low-similarity  score,  had 
considerable  change  across  the  same  time  period.  There  was  no  “gold-standard”  or  “ground- 
truth”  PFnet  available  to  compare  the  accuracy  of  both  teams’  post-PFnets. 
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Figure  15.  The  premeasure  PFnets  for  the  CPT-L  and  non-CPT-L  with  a  24%  similarity  between 
the  two  PFnets. 
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Figure  16.  The  postmeasure  PFnets  for  the  CPT-L  and  non-CPT-L  with  a  44%  similarity  between  the 
two  PFnets. 
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4.4.4  Qualitative  Pathfinder  Results 

Along  with  the  quantitative  analyses  that  have  been  presented  with  the  PFnets,  a  qualitative 
analysis  of  the  PFnets  can  be  done  to  examine  the  content  and  relationships  represented  by  the 
PFnets.  Basic  social  network  analyses  have  been  applied  within  the  PFnet  visualization  including 
the  concept  of  line  weights  representing  the  strength  of  the  connection  between  nodes  and  the 
node  size  representing  the  degree  centrality.  Degree  centrality  is  defined  as  the  number  of  nodes 
adjacent  to  a  given  node.  Concepts  that  are  highly  central  represent  a  core  or  high- importance 
element  for  the  overall  structure  or  situation.  For  example,  the  non-CPT-L  team  had  the 
“political  line  of  effort”  as  a  central  element  in  the  pre-PFnet  (figure  15b)  and  was  joined  by  the 
“political  economic  line  of  effort”  as  an  accompanying  central  concept  in  the  post-PFnet 
(figure  16b). 

Within  the  PFnets,  a  concept  can  be  focused  on  in  order  to  identify  how  the  individual  or  group 
is  defining  or  envisioning  that  concept  by  the  direct  relationships  existing  with  the  focus  concept 
(figure  17).  Once  the  concept  has  been  isolated,  it  can  be  easily  compared  to  that  of  other 
individuals  or  groups  for  differences  among  the  concept  focus. 


Figure  17.  Focus  on  the  concept  of  Political  economic  line  of  effort,  which  highlights  all  of  the  direct  linkages 
associated  with  this  concept. 
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Two  examples  of  different  perspectives  of  the  same  concept  between  the  CPT-L  and  non-CPT- 
L  teams  are  presented  here.  The  first  example  focuses  on  the  “political  economic  line  of  effort.” 
As  shown  in  figure  18,  the  two  teams  share  the  relationship  with  “economic  growth”  and  have  a 
link  to  another  line  of  effort.  However,  that  line  of  effort  is  different  between  the  two  teams — the 
CPT-L  team  had  a  relationship  between  the  “political  economic  line  of  effort”  and  the  “political 
line  of  effort,”  while  the  non-CPT-L  team  had  a  relationship  with  the  “rule  of  law  line  of  effort.” 
The  other  example  is  on  the  “regional  stability”  concept  where  the  non-CPT-L  team’s 
perspective  possesses  all  relationships  of  the  CPT-L  team  concerning  regional  stability  as  well 
as  two  others,  to  include  the  “political  line  of  effort”  and  “economic  growth”  (figure  19). 


Political  Economic  Line  of  Effort 

CPT-L  Team 

Concept  Focus 

Non-CPT-L  Team 

Oil  revenue  r- 

pRule  of  law  line  of  operalion 

Political  economic  line  A  ope ralfah^ 

Economic  growth 

^^Political  line  of  operation 

^""^PoliHeal  economic  line  of  operation 

^V^Economic  growth 

Figure  18.  Concept  focus  on  the  “political  economic  line  of  effort”  highlighting  the  differences  between  the  two 
teams. 


Regional 

Stability  Concept  Focus 

CPT-L  Team 

Non-CPT-L  Team 

Security  line  of  operation  m 

Regional  stability^ 

- 

Regional  slaonff^ 

Economic  growth  ^ 

Figure  19._Concept  focus  on  the  “regional  stability”  highlighting  the  differences  between  the  two  teams. 
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4.5  Shared  Understanding 


Pathfinder  similarity  measure  was  examined  among  team  members  to  quantitatively  assess  the 
shared  situation  awareness  between  team  members  of  the  same  team.  The  Pathfinder  similarity 
scores  between  every  team  member  were  averaged  for  the  pre  and  postmeasures  to  compute  the 
shared  understanding  within  a  team.  This  measure  of  interteam  congruency  provides  a 
quantitative  measure  of  shared  understanding  and  when  examined  over  time,  can  provide 
information  on  how  well  the  team  is  coming  together  or  drifting  apart  in  their  understanding  and 
awareness  of  the  situation.  As  expected,  given  equal  teams,  both  teams  scored  approximately  the 
same  on  the  premeasure  of  shared  situation  awareness  (figure  20).  However,  the  CPT-L  team 
again  showed  an  increase  over  time,  suggesting  that  the  CPT-L  team  members  are  becoming 
more  congruent  with  one  another  in  their  understanding  and  awareness  of  the  situation.  The  non- 
CPT-L  team,  on  the  other  hand,  shows  no  sign  of  improved  shared  situation  awareness  from  pre 
to  postmeasurement.  It  is  worth  noting  again  that  in  spite  of  the  unique  challenges  the  CPT-L 
team  faced,  they  were  able  to  show  slight  signs  of  improvement  over  the  planning  period. 
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Figure  20.  Pathfinder  interteam  similarity  scores  representing  team  shared  situation  awareness. 

4.6  Subjective  Data  Results 

Feedback  from  the  team  members  of  both  groups  were  collected  during  the  plenary  session  at  the 
end  of  the  day.  This  session  was  led  by  two  moderators  to  elicit  opinions,  experience  with  and 
without  the  CPT-L,  and  recommendations  for  improvement.  Most  plenary  comments  related  to 
usability  and  visualization  design  issues  with  the  CPT-L,  which  will  be  covered  more  fully  in 
Part  II:  CPT-L  Usability.  There  were  two  comments  concerning  the  appropriateness  of  an 
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experimental  design  for  this  event  considering  the  current  maturity  of  the  tool  and  the  amount  of 
training  provided.  It  was  expressed  that  given  the  early  development  stage  of  the  CPT-L,  it 
might  have  been  better  to  have  more  users  interacting  with  the  tool  and  providing  input  in  its  use 
over  an  experimental  and  control  design.  Many  participants,  particularly  from  participants  who 
used  the  CPT-L  during  planning,  expressed  optimism  in  the  potential  capabilities  of  the  tool  but 
felt  there  were  usability  issues  that  still  needed  to  be  resolved  before  that  potential  of  the  tool 
will  be  fully  realized. 


5.  Conclusions 


The  primary  experimental  hypothesis  of  this  LOE  was  that  the  CPT-L  would  effectively 
facilitate  team  planning  without  increasing  the  workload  of  its  users.  The  results  in  the  areas  of 
planning,  setting  objectives  and  alternatives,  and  situation  understanding  show  that  the  CPT-L 
was  as  effective — if  not  more  effective — in  facilitating  the  planning  process  compared  to  a  team 
using  more  traditional  tools  and  methods.  The  quality  of  the  planning  product  produced  could 
not  be  fairly  compared  due  to  the  confounds  of  the  experiment.  The  results  from  the  area  of  data 
manipulation  showed  that  there  was  no  increase  in  workload  across  any  of  the  six  dimensions 
measured  and  actually  was  significantly  lower  for  the  physical  workload  measure  compared  to 
the  non-CPT-L  team. 

The  early  developmental  state  of  the  CPT-L  appeared  to  create  more  of  a  user-testing  session 
than  a  pure-planning  session  for  the  CPT-L  team.  This  issue  was  compounded  with  the  limited 
and  noninteractive  training  provided  to  the  CPT-L  team  members.  These  issues  introduced 
challenges  to  the  CPT-L  team  that  were  not  intended  and  were  not  also  shared  by  the 
non-CPT-L  team.  Combined  with  fewer  team  members  than  the  non-CPT-L,  the  CPT-L  was 
placed  at  a  significant  disadvantage  compared  to  the  non-CPT-L  team.  Considering  these 
disadvantages,  it  is  therefore  surprising  that  the  CPT-L  team  displayed  converging  evidence  of 
higher  performance  and  greater  improvement  in  the  planning  process  compared  to  the 
non-CPT-L  team.  The  converging  evidence  ranging  from  the  reduced  workload,  higher  gains  in 
understanding,  and  slight  improvement  in  shared  situation  awareness  evident  in  the  CPT-L  team 
all  point  toward  the  possible  advantages  of  using  the  CPT-L.  Any  of  these  results  alone  does  not 
provide  definitive  results;  but  the  converging  nature  of  these  data  in  the  same  direction — 
especially  considering  the  disadvantages  mentioned  above — suggests  a  beneficial  role  of  the 
CPT-L  in  the  planning  process.  In  order  to  effectively  assess  the  impact  of  the  CPT-L  on  the 
product  of  the  planning  process,  additional  training  is  required  to  create  equal  expertise  with  the 
tools  being  used  across  both  teams  and  equal  amounts  of  planning  time  and  manpower  available 
is  required. 
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Appendix.  COMPOEX  Plan  Evaluation  Form 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Directions:  Please  review  both  Plan  A  &  Plan  B  provided.  Rate  each  plan  among  the  planning  dimensions  below  by 
placing  an  A  on  the  scale  for  Plan  A  and  a  B  on  the  same  scale  for  Plan  B.  If  the  exact  same  rating  is  given  for  both 
plans,  please  place  one  letter  on  the  appropriate  position  on  the  scale  and  the  other  letter  directly  above  it. 

Consistency:  Correspondence  or  agreement  among  the  parts  of  the  plan  from  the  Line  of  Effort  down  to  the  actions 


or  resources. 

1  1 

I  I  I 

1  1 

Inconsistent 

1  1  1  1 

Consistent 

Robustness:  The  resilience  of  the  plan  or  how  well  it  supports  planned  actions  before  replanning  is  needed. 

1 _ 1 _ 1 _ 1 _ 1 

1  1 

Low  robustness 

I  l  l  l 

High  robustness 

Conflicts:  The  degree  conflicts  are  present  among  the  planning  components  (e.g.,  across  Lines  of  efforts,  objectives, 

actions,  etc...) 

1  1 

l  l  l 

1  1 

Numerous  conflicts 

1  1  1  1 

No  conflicts 

Comprehensiveness:  Inclusion  of  many  relevant  planning  components  (e.g.,  across  Lines  of  efforts,  objectives, 
actions,  etc. . .)  both  in  breadth  (lines  of  efforts)  and  depth  (down  to  the  resource  and  node  level) 

1 _ 1 _ 1 _ 1 _ 1 

I  I 

Low  Comprehensiveness 

l  l  l  l 

High  Comprehensiveness 

Probability  for  plan  success:  The  judged  probabilistic  chance  of  success  of  the  plan 

1 _ 1 _ 1 _ 1 _ 1 

1  1 

Low  Success  Probability 

i  I  I  i 

High  Success  Probability 

Plan  Quality:  A  combination  of  all  plan  dimensions  above,  the  overall  evaluation  of  the  plans 

1 _ 1 _ 1 _ 1 _ 1 

1  1 

Low  Quality 

l  l  l  l 

High  Quality 
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Intentionally  left  blank. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


C2 

Command  and  Control 

COMPOEX 

Conflict  Modeling,  Planning,  and  Outcomes  Experimentation 

CPT-L 

Campaign  Planning  Tool-Light 

DARPA 

Defense  Advanced  Research  Project  Agency 

h 

hour  (s) 

IRR 

inter-rater  reliability 

JFCOM 

Joint  Forces  Command 

LOE 

limited  objective  experiment 

min 

minute  (s) 

NASA 

National  Aeronautics  and  Space  Administration 

PFnets 

Pathfinder  networks 

PMESII 

political,  economic,  social,  information,  and  infrastructure 

SME 

subject  matter  expert 

Sr. 

Senior 

TLX 

Task  Load  Index 
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